Fix OSV to handle affected_packages correctly & add support to collect commits #2080

ziadhany · 2025-12-23T01:38:55Z

Issue:

ziadhany · 2025-12-24T11:00:37Z

OSV Logs , importers:

pypa_importer_v2
pysec_importer_v2
oss_fuzz_importer_v2
github_osv_importer_v2

TG1999 · 2025-12-29T07:19:15Z

vulnerabilities/tests/test_data/osv_test/pypa/pypa-expected-1.json

+    {
+      "reference_id": "",
+      "reference_type": "",
+      "url": "http://git.fedorahosted.org/cgit/freeipa.git/commit/?id=a1991aeac19c3fec1fdd0d184c6760c90c9f9fc9"


Can we classify them as commit type ?

The problem is that this is passed as part of the reference URLs, so it’s not easy to detect whether it’s a commit URL or just an article. We have already classified some of these as reference_type=commit because they were passed as a Git range.

I think one option is to improve our capabilities to parse different commit URLs in the packageurl-python library and rely on improvers like pipelines/v2_improvers/collect_commits.py to handle this case.

ziadhany · 2025-12-31T05:55:52Z

pypa_importer_v2

from vulnerabilities.models import AdvisoryV2
from django.db.models import Count
duplicates = (
    AdvisoryV2.objects
    .values('avid')
    .annotate(count=Count('id'))
    .filter(count__gt=1)
)
len(duplicates)
Out[5]: 0
AdvisoryV2.objects.count()
Out[6]: 3227

pysec_v2.txt

from vulnerabilities.models import AdvisoryV2
from django.db.models import Count
duplicates = (
    AdvisoryV2.objects
    .values('avid')
    .annotate(count=Count('id'))
    .filter(count__gt=1)
)
len(duplicates)
Out[5]: 0
AdvisoryV2.objects.count()
Out[6]: 3289

oss_fuzz_v2 has a duplicated advisory_id ex:
https://github.com/search?q=repo%3Agoogle%2Foss-fuzz-vulns+OSV-2023-152&type=code
I’m not sure what the best option is in this case. Should we use the full path, like
vulns/https:/gitlab.com/wireshark/wireshark.git/OSV-2023-152.yaml as the advisory_id to avoid duplication?

from vulnerabilities.models import AdvisoryV2
from django.db.models import Count
duplicates = (
    AdvisoryV2.objects
    .values('avid')
    .annotate(count=Count('id'))
    .filter(count__gt=1)
)
len(duplicates)
Out[5]: 5
duplicates
Out[6]: <AdvisoryQuerySet [{'avid': 'oss_fuzz_importer_v2/OSV-2023-69', 'count': 2}, {'avid': 'oss_fuzz_importer_v2/OSV-2023-152', 'count': 2}, {'avid': 'oss_fuzz_importer_v2/OSV-2022-1108', 'count': 2}, {'avid': 'oss_fuzz_importer_v2/OSV-2023-38', 'count': 2}, {'avid': 'oss_fuzz_importer_v2/OSV-2023-49', 'count': 2}]>
AdvisoryV2.objects.count()
Out[7]: 3781

github_osv_importer_v2:

from vulnerabilities.models import AdvisoryV2
from django.db.models import Count
duplicates = (
    AdvisoryV2.objects
    .values('avid')
    .annotate(count=Count('id'))
    .filter(count__gt=1)
)
duplicates
Out[5]: <AdvisoryQuerySet []>
len(duplicates)
Out[6]: 0

vulnerabilities/tests/test_data/osv_test/oss-fuzz/oss-fuzz-1.yaml

vulnerabilities/models.py

ziadhany · 2025-12-31T08:26:15Z

For logs, we need to make a clear decision about which logs and data we ignore, and which ones we need to fix/support.

pysec_importer_v2:

ERROR 2025-12-31 05:03:11.688415 UTC Unsupported PyPI advisory data file: GHSA-227r-w5j2-6243.json

ERROR 2025-12-31 05:03:12.411085 UTC Unsupported PyPI advisory data file: MAL-2025-917.json

oss_fuzz_importer_v2

Unsupported package type: {'package': {'name': 'libbpf', 'ecosystem': 'OSS-Fuzz'}, 'ranges': [{'type': 'GIT', 'repo': 'https://github.com/libbpf/libbpf', 'events': [{'introduced': '421213a052aebb0c357b6d0872d6c57f2113800d'}, {'fixed': '741277511035893c72a34df05da3b943afa747a4'}]}], 'versions': ['v0.6.0', 'v0.6.1', 'v0.7.0', 'v0.8.0', 'v0.8.1', 'v1.0.0', 'v1.0.1'], 'ecosystem_specific': {'severity': 'HIGH'}} in OSV: 'OSV-2021-1576'

github_osv_importer_v2

Unsupported package type: {'package': {'ecosystem': 'Go', 'name': 'github.com/cortexproject/cortex'}, 'ranges': [{'type': 'ECOSYSTEM', 'events': [{'introduced': '0'}, {'last_affected': '1.9.0'}]}]} in OSV: 'GHSA-jphm-g89m-v42p'

(This appears to be a bug.) Unsupported package type: {'package': {'ecosystem': 'crates.io', 'name': 'deno'}, 'ranges': [{'type': 'ECOSYSTEM', 'events': [{'introduced': '1.5.0'}, {'fixed': '1.10.2'}]}], 'database_specific': {'last_known_affected_version_range': '<= 1.10.1'}} in OSV: 'GHSA-xpwj-7v8q-mcgj'

Unsupported package type: {'package': {'ecosystem': 'Pub', 'name': 'archive'}, 'ranges': [{'type': 'ECOSYSTEM', 'events': [{'introduced': '0'}, {'fixed': '3.3.8'}]}], 'database_specific': {'last_known_affected_version_range': '<= 3.3.7'}} in OSV: 'GHSA-r285-q736-9v95'

Unsupported package type: {'package': {'ecosystem': 'SwiftURL', 'name': 'github.com/marmelroy/Zip'}, 'ranges': [{'type': 'ECOSYSTEM', 'events': [{'introduced': '0'}, {'last_affected': '2.1.2'}]}]} in OSV: 'GHSA-g454-wj9r-jpg4'

Add support to collect commits Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Use parse_advisory_data_v3 for GitHub OSV. Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Update the function docs osv_v2 Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Fix CVSSv4 vector length issue Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Signed-off-by: ziad hany <ziadhany2016@gmail.com>

…es to keep ranges consistent. Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Signed-off-by: ziad hany <ziadhany2016@gmail.com>

keshav-space

Thanks @ziadhany, see some comments below.

Regarding pysec_importer_v2 log it is very vague. It gives no information about what the problem with the advisory. Is it that we can’t parse the version, package is missing, or something else? Log should be actionable.

vulnerabilities/tests/test_data/osv_test/github/github-expected-2.json

vulnerabilities/tests/test_data/osv_test/github/github-expected-3.json

ziadhany · 2026-01-13T15:57:52Z

@keshav-space The problem is that in OSV we have three types of version ranges:

Explicit version list

"versions": [
  "2.8.0", "2.8.0.post1", "2.8.0.post2",
  "2.9.0", "2.9.1", "2.9.2"
]

Range-based version

{
  "type": "ECOSYSTEM",
  "events": [
    { "introduced": "2.8.0" },
    { "fixed": "2.10.0" }
  ]
}

last_known_affected_version_range

"database_specific": {
  "last_known_affected_version_range": "<= 0.21.7"
}

We cannot merge these, because doing so would produce an invalid version constraint.

For example:
https://github.com/github/advisory-database/blob/8a993f4259457ec9f447777c053316da28eb5908/advisories/github-reviewed/2020/08/GHSA-wwrj-35w6-77ff/GHSA-wwrj-35w6-77ff.json#L9

This results in the following error:

univers.version_constraint.InvalidConstraintsError:
Invalid constraints sequence: ('< 1.0.5', '< 1.0.7')
in [
  VersionConstraint(comparator='<', version=SemverVersion(string='1.0.5')),
  VersionConstraint(comparator='<', version=SemverVersion(string='1.0.7'))
]

Logs from the old pipeline when I tried to merge them:

osv_1.zip

keshav-space · 2026-01-13T16:14:31Z

@ziadhany

Let’s take this osv range as an example here:

{
   "package": {
      "ecosystem": "Packagist",
      "name": "devcode-it/openstamanager"
   },
   "ranges": [
      {
         "type": "ECOSYSTEM",
         "events": [
            {
               "introduced": "0"
            },
            {
               "fixed": "2.9.5"
            }
         ]
      }
   ],
   "database_specific": {
      "last_known_affected_version_range": "<= 2.9.4"
   }
}

This is the right set of affected package for the above range.

[
   {
      "package": "pkg:composer/devcode-it/openstamanager",
      "affected_version_range": "vers:composer/<=2.9.4",
      "fixed_version_range": "vers:composer/2.9.5",
      "introduced_by_commit_patches": [],
      "fixed_by_commit_patches": []
   }
]

This is set of affected packages that we're getting currently, which is not accurate.

[
   {
      "package": "pkg:composer/devcode-it/openstamanager",
      "affected_version_range": "vers:composer/<2.9.5",
      "fixed_version_range": "vers:composer/2.9.5",
      "introduced_by_commit_patches": [],
      "fixed_by_commit_patches": []
   },
   {
      "package": "pkg:composer/devcode-it/openstamanager",
      "affected_version_range": "vers:composer/<=2.9.4",
      "fixed_version_range": null,
      "introduced_by_commit_patches": [],
      "fixed_by_commit_patches": []
   }
]

…en last_known_affected_version_range, and finally range-based versions Signed-off-by: ziad hany <ziadhany2016@gmail.com>

ziadhany force-pushed the osv-migration branch from 91b89f3 to ab084af Compare December 24, 2025 07:16

ziadhany marked this pull request as ready for review December 24, 2025 10:47

ziadhany requested review from TG1999 and keshav-space December 28, 2025 15:54

ziadhany force-pushed the osv-migration branch from ebaee7d to b6bf165 Compare December 28, 2025 16:16

TG1999 reviewed Dec 29, 2025

View reviewed changes

ziadhany commented Dec 31, 2025

View reviewed changes

vulnerabilities/tests/test_data/osv_test/oss-fuzz/oss-fuzz-1.yaml Show resolved Hide resolved

ziadhany commented Dec 31, 2025

View reviewed changes

vulnerabilities/models.py Outdated Show resolved Hide resolved

ziadhany added 10 commits January 1, 2026 18:32

Fix OSV to handel affected_packages correctly

07edf7d

Add support to collect commits Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Add more tests.

85f8771

Use parse_advisory_data_v3 for GitHub OSV. Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Fix the test for osv pipelines

f06fe18

Update the function docs osv_v2 Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Fix the test for osv pipelines

308917f

Fix CVSSv4 vector length issue Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Update OSV migration file

0108c01

Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Update OSV to simplify the affected/fixed package constraints

5b95c3a

Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Avoid mixing explicit affected packages with last known affected rang…

142dbc0

…es to keep ranges consistent. Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Update OSV to catch error constraints correctly

f911eea

Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Add a test to ensure invalid versions are skipped

d11e07d

Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Add a test to ensure that OSS-Fuzz uses the explicit versions correctly

45a43c0

Signed-off-by: ziad hany <ziadhany2016@gmail.com>

ziadhany force-pushed the osv-migration branch from fa1dfa2 to 45a43c0 Compare January 1, 2026 16:34

Drop old migration file

8ec85fa

Signed-off-by: ziad hany <ziadhany2016@gmail.com>

keshav-space requested changes Jan 12, 2026

View reviewed changes

vulnerabilities/tests/test_data/osv_test/github/github-expected-2.json Outdated Show resolved Hide resolved

vulnerabilities/tests/test_data/osv_test/github/github-expected-3.json Outdated Show resolved Hide resolved

Update OSV affected_version_range to prioritize explicit versions, th…

8a2e0e8

…en last_known_affected_version_range, and finally range-based versions Signed-off-by: ziad hany <ziadhany2016@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Fix OSV to handle affected_packages correctly & add support to collect commits #2080

Fix OSV to handle affected_packages correctly & add support to collect commits #2080

Uh oh!

ziadhany commented Dec 23, 2025 •

edited

Loading

Uh oh!

ziadhany commented Dec 24, 2025 •

edited

Loading

Uh oh!

TG1999 Dec 29, 2025

Uh oh!

ziadhany Dec 29, 2025 •

edited

Loading

Uh oh!

ziadhany commented Dec 31, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

ziadhany commented Dec 31, 2025 •

edited

Loading

Uh oh!

keshav-space left a comment

Uh oh!

Uh oh!

Uh oh!

ziadhany commented Jan 13, 2026 •

edited

Loading

Uh oh!

keshav-space commented Jan 13, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Fix OSV to handle affected_packages correctly & add support to collect commits #2080

Are you sure you want to change the base?

Fix OSV to handle affected_packages correctly & add support to collect commits #2080

Uh oh!

Conversation

ziadhany commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ziadhany commented Dec 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TG1999 Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

ziadhany Dec 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ziadhany commented Dec 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ziadhany commented Dec 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

keshav-space left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ziadhany commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

keshav-space commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ziadhany commented Dec 23, 2025 •

edited

Loading

ziadhany commented Dec 24, 2025 •

edited

Loading

ziadhany Dec 29, 2025 •

edited

Loading

ziadhany commented Dec 31, 2025 •

edited

Loading

ziadhany commented Dec 31, 2025 •

edited

Loading

ziadhany commented Jan 13, 2026 •

edited

Loading

keshav-space commented Jan 13, 2026 •

edited

Loading